[RNE Rewrite] feat: Add OCR implementation - #1322
Draft
benITo47 wants to merge 21 commits into
Draft
Conversation
…TC, rectifyQuad, warpByGrid, rotate)
…essive OCR screen
…OcrOptions.tables) Tables were the odd one out — orientation/dewarp had per-run toggles, tables fired unconditionally when a table region was found. Add RunOcrOptions.tables (default true when the model supports them; a per-run false skips the recognizer), gate the table pass on it, and surface a Table-structure sub-toggle in the demo alongside orientation/dewarp.
…ailing | _ glyphs)
Flat-gray strip padding left a content→pad seam EasyOCR's CRNN read as a
trailing glyph. Add OcrModelOptions.recognizerPadMode ('constant' default,
'cornerMean' = background-matched), thread it through the recognizer, default
EasyOCR to cornerMean. Not a decode bug — the argmax was predicting the seam.
…f composed strip Composing a vertical stack into one horizontal strip fed the recognizer a non-word column with its horizontal-word context, making it hallucinate words (and the vertical-digit phantoms). Recognize each glyph on its own and join top-to-bottom — drops the word bias, gives per-glyph confidence. Cost: one recognizer pass per glyph (fine for the opt-in vertical path).
Vertical reads skip the horizontal drop-score gate because they run lower-confidence. Expose RunOcrOptions.verticalMinConfidence so callers can raise it to filter the low-confidence hallucinations the vertical path emits.
Hardens the pipeline — encodeConstraint variant handling, snapUpDim lattice clamp, span-aware table fill, tagged shape-constraint union, dead compose-mode removal, JSI length guards and demo polish. Also pins the document pipeline to the now-published 0.9.0 model tag.
…chema contract Replaces the removed get_dynamic_dims_/get_enum_shapes_ companion API with validateSpec over model.schema: - detect/recognize input sizes are now per-dimension ConcreteDim domains, so one snap routine covers constant/range/enum and the CoreML enumerated grid needs no cross-dimension shape set. - the recognizer's width-to-CTC-timestep relation is read from the linear runtime constraint the model declares instead of inferred as a width/timestep ratio, which was wrong for EasyOCR's CRNN (512 = 4*127 + 4, not a multiple). - TextBoxExtractor declares the detect output layout it decodes, so the allowed spec covers the outputs instead of wildcarding them. - documentModels validates its four static methods in one spec; repeated symbols now enforce the vocab/hidden/feature agreements the manual checks did. - useOcr resolves its nested models through the single-pass resource download.
…HF repos The four OCR repos now follow the MODEL_SPEC layout the rest of the ecosystem uses — per-backend directories, a root config.json, and <model>[_<size>]_<backend>_<precision>.pte file names — published under v0.10.0, so they share NEXT_VERSION_TAG with the other rewrite models instead of the unprefixed 0.9.0 tag they had. Precision is now part of the path, so each preset spells out what it actually ships rather than hiding it behind a bare backend name.
PP-DocLayoutV3's RT-DETR head is set prediction, so the preset runs it with NMS off — but it still emits nested duplicates, e.g. a region for one line inside the region for the paragraph that contains it. Every region is OCR'd on its own crop, so those duplicates transcribed the same text twice: a page with "Lekcja / Indywidualna" came back as three blocks, one per nesting. IoU suppression cannot fix this (a small box inside a large one has low IoU), so the regions are now merged on containment, as PaddleX's `layout_merge_bboxes_mode: 'large'` does: largest first, drop any region a kept one already covers by >=80% of its own area. Visual regions never suppress, so a caption inside a figure is still read. It also stops recognizing those crops twice.
…very OCR backend The OCR and Model Inspector screens never read the safe-area insets the other eight screens already pad by, so "Run OCR" sat under the navigation bar, and the main menu was a fixed View whose tenth button was clipped by it. The menu scrolls now, still centered while it fits. The OCR screen also filtered its model list by platform instead of disabling what the platform cannot run, so CoreML was invisible on Android. All six variants are listed now, with CoreML disabled off iOS and Vulkan disabled on it, matching every other screen.
Brings the OCR pipeline in line with the conventions that landed on rne-rewrite while this branch was open. - The three OCR-specific native ops were called straight off `rnexecutorchJsi.cv` from inside the task files. add-native-extension requires task-specific ops to be wrapped in `src/extensions/<domain>/utils/<name>.ts` and re-exported from the domain index, the way vadUtils/supertonicUtils/privacyFilterUtils are, so power users can reach them: new `cv/utils/ocrUtils.ts` with typed, documented `extractCraftTextBoxes` / `extractDbnetTextBoxes` / `ctcGreedyDecode`. - Five bare `throw new Error` (geometry, detectors, documentModels, quad ops) now carry codes: INVALID_ARGUMENT for a caller's bad input, INVALID_STATE for calling table recognition on models loaded without a table config. Documented with `@throws` on the entry points that can raise them. - `opts` parameters renamed to `options` (#1358): `makeCraftExtractBoxes`, `makeDbnetExtractBoxes`, `rectifyQuad`. - `OcrModel.ocrOpts` -> `modelOpts`, matching every other task's model type. - Tensor locals carry the `t` prefix the pipeline skill asks for.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds a unified, OCR and document understanding pipeline to react-native-executorch. Two OCRs - EasyOCR and PP-OCRv6, plus a higher-level document pipeline that orchestrates orientation correction, UVDoc dewarp, PP-DocLayoutV3 region layout, SLANet table-structure recognition, and reading-order assembly into HTML.
Introduces a breaking change?
Type of change
Tested on
Testing instructions
Try OCRing on different images, use different OCR options. See if anything is ill behaved.
Screenshots
Related issues
Checklist
Additional notes